Multi-task learning for the prediction of wind power ramp events with deep neural networks.

In Machine Learning, the most common way to address a given problem is to optimize an error measure by training a single model to solve the desired task. However, sometimes it is possible to exploit latent information from other related tasks to improve the performance of the main one, resulting in a learning paradigm known as Multi-Task Learning (MTL). In this context, the high computational capacity of deep neural networks (DNN) can be combined with the improved generalization performance of MTL, by designing independent output layers for every task and including a shared representation for them. In this paper we exploit this theoretical framework on a problem related to Wind Power Ramps Events (WPREs) prediction in wind farms. Wind energy is one of the fastest growing industries in the world, with potential global spreading and deep penetration in developed and developing countries. One of the main issues with the majority of renewable energy resources is their intrinsic intermittency, which makes it difficult to increase the penetration of these technologies into the energetic mix. In this case, we focus on the specific problem of WPREs prediction, which deeply affect the wind speed and power prediction, and they are also related to different turbines damages. Specifically, we exploit the fact that WPREs are spatially-related events, in such a way that predicting the occurrence of WPREs in different wind farms can be taken as related tasks, even when the wind farms are far away from each other. We propose a DNN-MTL architecture, receiving inputs from all the wind farms at the same time to predict WPREs simultaneously in each of the farms locations. The architecture includes some shared layers to learn a common representation for the information from all the wind farms, and it also includes some specification layers, which refine the representation to match the specific characteristics of each location. Finally we modified the Adam optimization algorithm for dealing with imbalanced data, adding costs which are updated dynamically depending on the worst classified class. We compare the proposal against a baseline approach based on building three different independent models (one for each wind farm considered), and against a state-of-the-art reservoir computing approach. The DNN-MTL proposal achieves very good performance in WPREs prediction, obtaining a good balance for all the classes included in the problem (negative ramp, no ramp and positive ramp).

The neutron camera upgrade for MAST Upgrade.

The Neutron Camera Upgrade (NCU) is a neutron flux monitor consisting of six lines of sight (LoSs) under installation on Mega Ampere Spherical Tokamak (MAST) Upgrade. The NCU is expected to contribute to the study of the confinement of fast ions and on the efficiency of non-inductive current drive in the presence of on-axis and off-axis neutral beam injection by measuring the neutron emissivity profile along the equatorial plane. This paper discusses the NCU main design criteria, the engineering and interfacing issues, and the solutions adopted. In addition, the results from the characterization and performance studies of the neutron detectors using standard γ-rays sources and a 252Cf source are discussed. The proposed design has a time resolution of 1 ms with a statistical uncertainty of less than 10% for all MAST Upgrade scenarios with a spatial resolution of 10 cm: higher spatial resolution is possible by moving the LoSs in-between plasma discharges. The energy resolution of the neutron detector is better than 10% for a light output of 0.8 MeVee, and the measured pulse shape discrimination is satisfactory.

Compositional generative mapping for tree-structured data--part II: topographic projection model.

We introduce GTM-SD (Generative Topographic Mapping for Structured Data), which is the first compositional generative model for topographic mapping of tree-structured data. GTM-SD exploits a scalable bottom-up hidden-tree Markov model that was introduced in Part I of this paper to achieve a recursive topographic mapping of hierarchical information. The proposed model allows efficient exploitation of contextual information from shared substructures by a recursive upward propagation on the tree structure which distributes substructure information across the topographic map. Compared to its noncompositional generative counterpart, GTM-SD is shown to allow the topographic mapping of the full sample tree, which includes a projection onto the lattice of all the distinct subtrees rooted in each of its nodes. Experimental results show that the continuous projection space generated by the smooth topographic mapping of GTM-SD yields a finer grained discrimination of the sample structures with respect to the state-of-the-art recursive neural network approach.

A self-organizing map for adaptive processing of structured data.

Recent developments in the area of neural networks produced models capable of dealing with structured data. Here, we propose the first fully unsupervised model, namely an extension of traditional self-organizing maps (SOMs), for the processing of labeled directed acyclic graphs (DAGs). The extension is obtained by using the unfolding procedure adopted in recurrent and recursive neural networks, with the replicated neurons in the unfolded network comprising of a full SOM. This approach enables the discovery of similarities among objects including vectors consisting of numerical data. The capabilities of the model are analyzed in detail by utilizing a relatively large data set taken from an artificial benchmark problem involving visual patterns encoded as labeled DAGs. The experimental results demonstrate clearly that the proposed model is capable of exploiting both information conveyed in the labels attached to each node of the input DAGs and information encoded in the DAG topology.

Analysis of the internal representations developed by neural networks for structures applied to quantitative structure--activity relationship studies of benzodiazepines.

An application of recursive cascade correlation (CC) neural networks to quantitative structure-activity relationship (QSAR) studies is presented, with emphasis on the study of the internal representations developed by the neural networks. Recursive CC is a neural network model recently proposed for the processing of structured data. It allows the direct handling of chemical compounds as labeled ordered directed graphs, and constitutes a novel approach to QSAR. The adopted representation of molecular structure captures, in a quite general and flexible way, significant topological aspects and chemical functionalities for each specific class of molecules showing a particular chemical reactivity or biological activity. A class of 1,4-benzodiazepin-2-ones is analyzed by the proposed approach. It compares favorably versus the traditional QSAR treatment based on equations. To show the ability of the model in capturing most of the structural features that account for the biological activity, the internal representations developed by the networks are analyzed by principal component analysis. This analysis shows that the networks are able to discover relevant structural features just on the basis of the association between the molecular morphology and the target property (affinity).

Discriminant pattern recognition using transformation-invariant neurons.

To overcome the problem of invariant pattern recognition, Simard, LeCun, and Denker (1993) proposed a successful nearest-neighbor approach based on tangent distance, attaining state-of-the-art accuracy. Since this approach needs great computational and memory effort, Hastie, Simard, and Säckinger (1995) proposed an algorithm (HSS) based on singular value decomposition (SVD), for the generation of nondiscriminant tangent models. In this article we propose a different approach, based on a gradient-descent constructive algorithm, called TD-Neuron, that develops discriminant models. We present as well comparative results of our constructive algorithm versus HSS and learning vector quantization (LVQ) algorithms. Specifically, we tested the HSS algorithm using both the original version based on the two-sided tangent distance and a new version based on the one-sided tangent distance. Empirical results over the NIST-3 database show that the TD-Neuron is superior to both SVD- and LVQ-based algorithms, since it reaches a better trade-off between error and rejection.

On the implementation of frontier-to-root tree automata in recursive neural networks.

In this paper we explore the node complexity of recursive neural network implementations of frontier-to-root tree automata (FRA). Specifically, we show that an FRAO (Mealy version) with m states, l input-output labels, and maximum rank N can be implemented by a recursive neural network with O(radical(log l+log m)lm(N)/log l+N log m) units and four computational layers, i.e., without counting the input layer. A lower bound is derived which is tight when no restrictions are placed on the number of layers. Moreover, we present a construction with three computational layers having node complexity of O((log l + log m)radical lmN) and O((log l + log m) lmN) connections. A construction with two computational layers is given that implements any given FRAO with a node complexity of O(lmN) and O((log l+N log m)lmN) connections. As a corollary we also get a new upper bound for the implementation of finite-state automata (FSA) into recurrent neural networks with three computational layers.

A general framework for adaptive processing of data structures.

A structured organization of information is typically required by symbolic processing. On the other hand, most connectionist models assume that data are organized according to relatively poor structures, like arrays or sequences. The framework described in this paper is an attempt to unify adaptive models like artificial neural nets and belief nets for the problem of processing structured information. In particular, relations between data variables are expressed by directed acyclic graphs, where both numerical and categorical values coexist. The general framework proposed in this paper can be regarded as an extension of both recurrent neural networks and hidden Markov models to the case of acyclic graphs. In particular we study the supervised learning problem as the problem of learning transductions from an input structured space to an output structured space, where transductions are assumed to admit a recursive hidden statespace representation. We introduce a graphical formalism for representing this class of adaptive transductions by means of recursive networks, i.e., cyclic graphs where nodes are labeled by variables and edges are labeled by generalized delay elements. This representation makes it possible to incorporate the symbolic and subsymbolic nature of data. Structures are processed by unfolding the recursive network into an acyclic graph called encoding network. In so doing, inference and learning algorithms can be easily inherited from the corresponding algorithms for artificial neural networks or probabilistic graphical model.

Supervised neural networks for the classification of structures.

Standard neural networks and statistical methods are usually believed to be inadequate when dealing with complex structures because of their feature-based approach. In fact, feature-based approaches usually fail to give satisfactory solutions because of the sensitivity of the approach to the a priori selection of the features, and the incapacity to represent any specific information on the relationships among the components of the structures. However, we show that neural networks can, in fact, represent and classify structured patterns. The key idea underpinning our approach is the use of the so called "generalized recursive neuron", which is essentially a generalization to structures of a recurrent neuron. By using generalized recursive neurons, all the supervised networks developed for the classification of sequences, such as backpropagation through time networks, real-time recurrent networks, simple recurrent networks, recurrent cascade correlation networks, and neural trees can, on the whole, be generalized to structures. The results obtained by some of the above networks (with generalized recursive neurons) on the classification of logic terms are presented.

Stability properties of labeling recursive auto-associative memory.

Labeling recursive auto-associative memory (LRAAM) is an extension of the RAAM model by Pollack (1990) to obtain distributed reduced representations of labeled directed graphs. In this paper some mathematical properties of LRAAM are discussed. Specifically, sufficient conditions on the asymptotical stability of the decoding process along a cycle of the encoded structure are given. LRAAM can be transformed into an analog Hopfield network with hidden units and an asymmetric connections matrix by connecting the output units with the input units. In this architecture encoded data can be accessed by content and different access procedures can be defined depending on the access key. Each access procedure corresponds to a particular constrained version of the recurrent network. The authors give sufficient conditions under which the property of asymptotical stability of a fixed point in one particular constrained version of the recurrent network can be extended to related fixed points in different constrained versions of the network. An example of encoding of a labeled directed graph on which the theoretical results are applied is given and discussed.

Intra- and interobserver concordance in scoring Harris lines: a test on bone sections and radiographs.

Little attention has been devoted to assessing the reproducibility of (paleo) pathological observations. Harris lines (HL) are among the markers most used to determine chronology of stresses suffered during growth. Nevertheless, their scoring entails remarkable methodological difficulty. Bone sections (S) and radiographs (R) of 29 adult tibiae of archeological provenance (medieval) were scored for HL by five observers. At regular intervals of time, each observer gave two independent counts on both series. Results show a) a substantial interobserver disagreement of HL estimates for both sectional and radiographic records, and b) a high level of intraobserver error.

Behavior-induced auditory exostoses in imperial Roman society: evidence from coeval urban and rural communities near Rome.

Presence and features of auditory exostoses were investigated in two cranial samples of Roman imperial age (1st-3rd century A.D.). The skeletal material comes from the necropolises of Portus (Isola Sacra) and Lucus Feroniae (Via Capenate), two towns along the Tevere River, in close relation with the social and economic life of Rome. Deep-rooted differences between the human communities represented by the skeletal samples (83 and 71 individuals, respectively, in this study) are documented both historically and archaeologically. The results show lack of exostoses in the female sex, a negligible incidence among the males of Lucus Feroniae, but a high frequency in the male sample from Isola Sacra (31.3%). Auditory exostoses are commonly recognised as localized hyperplastic growths of predominantly acquired origin. Features of the exostoses found in the male crania from Isola Sacra (particularly in relation to the age at death of the affected individuals) support this view. Furthermore, several clinical and anthropological studies have pointed out close links between the occurrence of auditory exostoses and prolonged cold water exposure, generally due to the practice of aquatic sports, or to working activities involving water contact or diving. In this perspective, the differences observed between the two Roman populations and between the sexes (in Isola Sacra) appear to result from different social habits: the middle class population of Portus habitually used thermal baths, whereas it is probable that thermae were seldom frequented (if at all) by the Lucus Feroniae population represented in the necropolis (mostly composed by slaves or freedmen farm laborers).(ABSTRACT TRUNCATED AT 250 WORDS)